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(54) Method oi incorporating a variable rate auxiliary data stream with a variable rate primary 
data stream 

(57) A method of incorporating an auxiliary signal with an input video signal includes the step of encoding the input 
video signal to generate a variable bit rate encoded signal having a selected level of distortion. The video signal is 
applied to a buffer. The level of distortion is selected as a function of the fullness of the buffer. The video signal is read 
from the buffer and applied to a transmission channel at a predetermined video output rate. Portions of an auxiliary 
signal are applied to the channel at times when the buffer is in an underflow condition. The selected level of distortion 
has a flow which corresponds to a predetermined, non-zero level of perceptual distortion in the video signal. 
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Description 

Background of the Invention 

The present invention relates to the transmission of 
primary and auxiliary digital data streams over a broad- 
cast channel. 

Systems, such as HDTV systems, use encoding 
techniques to extract redundancies from video signals 
which are transmitted as video frames from a transmitter 
to a receiver The amount of information contained in the 
encoded video signals tends to be variable, resulting in 
a bursty signal. For example, scenes involving a lot of 
action, such as basketball footage, contain less 
frame-to-frame redundant information, and therefore 
tend to produce a larger amount of encoded information 
or data, than the amount of encoded data required to 
transmit subsequent frames of a still picture, which con- 
tain primarily redundant information. 

For transmission using a limited-bit-rate channel, an 
HDTV encoder must produce a constrained constant bit 
rate. In order to ensure a constant bit rate output, a buffer 
is associated with the HDTV transmitter which accumu- 
lates the bursty video" signals and controls the rate at 
which the bitstream is transmitted. If there is a break in 
the transm ission of the bitstream generated by the H DTV 
transmitter such that the buffer becomes empty, partial 
or total loss of picture results. 

In an HDTV system, video data are received from a 
data source and compressed by a video encoder Among 
other things, the video encoder determines the level, i.e. 
coarseness of quantization of the video data stream. The 
video data are then transmitted to a buffer. The buffer 
controls the output rate of the video data stream. The 
fullness of the buffer is controlled by distortion control via 
a feedback mechanism. The distortion control controls 
the level of distortion, i.e., the coarseness or fineness of 
the video data, for each video frame which, in turn, ef- 
fects the level of, i.e.. makes more coarse, the quantiza- 
tion of the encoded video data. When the buffer fills up 
to a point above some reference level, the distortion con- 
trol causes the encoder to increase the level of quanti- 
zation of the video data stream, thereby reducing buffer 
fullness. When the buffer fullness falls below some ref- 
erence level, the distortion control causes the encoder 
to reduce the level of quantization of the video data 
stream, thereby increasing buffer fullness. In the event 
of buffer underflow, i.e. , so many more bits leave the buff- 
er than enter the buffer that the buffer empties. 

Summary of the Invention 

In the prior art. the buffer as a practical matter rarely 
empties even when a highly redundant image, such as 
a still image, is being sent. The reason is that even a still 
image contains noise which gives rise to some 
frame -to-frame differences which cause the encoder to 
increase the level of quantization. 



We have recognized that by establishing a floor on 
the allowed level of distortion, the buffer tends to more 
frequently realize an underflow condition. Auxiliary data 
representative of data services, such as electronic pub- 

5 lications. can be transmitted during the underflow condi- 
tion. While having a floor on the distortion level tends to 
increase the level of quantization of the video data 
stream compared to what a prior art system would do 
resulting in reduced picture quality over that which the 

10 prior art could achieve, the floor is established at such a 
level that the picture quality is acceptable to the human 
eye. 

Brief Description of the Drawing 

is 

FIG. 1 illustrates a schematic diagram of an HDTV 
system implemented in accordance with the present in- 
vention. 

FIG. 2 illustrates a block diagram of the HDTVtrans- 
20 milter of the HDTV system of FIG. 1 . 

FIG. 3 illustrates a block diagram of the HDTV re- 
ceiver of the HDTV system of FIG. 1 . 

FIG. 4 illustrates a graph depicting a measurement 
of the distortion level over time t. 
25 FIG. 5 illustrates a graph depicting the relationship 

between the level of distortion and bit rate. 

Detailed Description 

30 A general overview of a video system, illustratively 
a high definition television (HDTV) system 10, incorpo- 
rating the principles of the present invention is illustrated 
in FIG. 1 . The details of an HDTV video system are de- 
scribed in such U.S. Patents as 5,144.423 entitled 

35 "HDTV Encoder with Forward Estimation and Constant 
Rate Motion Vectors" issued September 1, 1992 to 
Knauer et al.: 5,136.377 entitled "Adaptive Non-linear 
Quantizer", issued August 4, 1992 to Johnston et al.: 
5,134,475 entitled "Adaptive Leak HDTV Encoder", is- 

40 sued July 28, 1 992 to Johnston et al.; and 5, 1 34,477 en- 
titled "HDTV Receiver" issued July 28, 1992 to Knauer 
et al. 

In operation, a viewer sends instructions to a central 
data server 1 2 to identify one or more data services that 

45 are desired to be received The instructions are transmit- 
ted from the viewer's remote location to the central serv- 
er 12 by means of a cable or fiber optic network 14. For 
example, the instructions might be transmitted as in- 
fra-red signals via a remote control unit 16 to a set-top 

so box 18 associated with receiver 30 which transmits the 
appropriate signals over the network 14. Alternatively, 
the viewer could transmit his instructions to central serv- 
er 1 2 over a standard telephone network (not shown). A 
separate data service converter box (not shown) may be 

55 used in place of the set-top box 1 8 to transmit the instruc- 
tions over network 14. 

The central server 12 processes the incoming re- 
quests from the viewers and retrieves the data desired 
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by the viewers from a respective one of servers 24. The 
data services can be any type of video or audio informa- 
tion such as. but not limited to, electronic newspapers, 
magazines, catalogs, movies transmitted to a storage 
device, or other types of services which are not required 
to be transmitted in real time. The data is transmitted to 
the set-top box 18. The set-top box 18 includes storage 
means, such as memory, for storing the data. Once a 
significant portion of the data has been transmitted to the 
set-top box 1 8, the service can be viewed by the viewer. 
Set-top box also receives video signals over network 1 4 
from a central HDTV transmitter 20. 

While the data is being transmitted to the viewer's 
set-top box 18, the viewer is still capable of viewing pro- 
grams broadcast from the HDTV transmitter 20. In ac- 
cordance with the present invention, HDTV video signals 
and the data signals are multiplexed together in such a 
manner that the HDTV video signals are transmitted in 
real time and the data service signals are transmitted to 
the set-top box at a slower rate. As such : the HDTV sig- 
nals are the primary signals broadcast to the set-top box 
18 and the data signals are treated as auxiliary signals 
as will be described in detail hereinafter. 

FIG. 2 is a moredetailed block diagram of the HDTV 
transmitter 20 shown in FIG. 1 . A high-definition (HD) vid- 
eo signal on lead 205 is encoded by an HDTV encoder 
210. As is well known, an HDTV encoder-indeed, most 
video encoders —provides a variable-bit-rate output de- 
pending on the nature of the video signal being encoded. 
For example, a video signal which is changing by a rel- 
atively small amount from frame to frame can be encod- 
ed with fewer bits than when the frame to frame differ- 
ences are relatively great, as when there is a lot of motion 
in the image. A sharp scene change also causes the en- 
coder output bit rate to increase for a time. However, 
while small scene changes require fewer bits than drastic 
scene changes, noise present in the video signals caus- 
es the image to become distorted. As a result, the 
number of bits required to encode the scene changes is 
significant. 

As a result of the above-described variability in the 
output bit rate of encoder 210. its output is buffered in a 
video data buffer 21 5 so that an appropriately formatted 
HDTV signal can be developed and applied to the broad- 
cast channel 220. Interposed between video data buffer 
21 5 and channel 220 is a multiplexer 225 whose function 
will be explained momentarily. 

The system also includes distortion control 230 
which receives as its inputs the incoming video signal 
from lead 205 as well as a "buffer fullness" signal on lead 
235 provided by buffer 215. The function and structure 
of distortion control 230 are well known in the art. The 
distortion control 230 determines the degree of coarse- 
ness to which a video signal is encoded-its distortion lev- 
el-taking into consideration factors, such as, but not lim- 
ited to, discrete cosine transform (DCT) coefficients, mo- 
tion vectors and the level of quantization. The coded vid- 
eo signal must contain enough information so that the 



coded image maintains the integrity of the original image. 
However, that measurement of integrity is subject to the 
constraint that the coded image will be viewed by a hu- 
man observer who will not be aware of small distortions 
5 in the coded image. As long as the human observer can 
not detect distortions resulting from the coding of the im- 
age, the distortion level is considered to be within an ac- 
ceptable range. 

The distortion level is dependent on a perceptual 
to distortion metric. The perceptual distortion metric pro- 
vides a measurement by which the degree of fineness 
to which the video signals are encoded may be deter- 
mined so that the distortion present in the coded image 
appears, to a human observer to be uniformly distribut- 
es ed. In accordance with the present invention, the percep- 
tual distortion metric is set at such a level that a floor is 
established for the distortion level. Thus, while the image 
may be capable of being more finely encoded, such en- 
coding would not be perceptible to the human observer. 
20 The relationship between the distortion level and the 
bit rate of the encoded video signal is illustrated in FIG. 
5. The distortion level and the bit rate have a generally 
inverse relationship. When the distortion level is high, the 
rate is low. When the distortion level is low, the rate is 
2S high. By setting a perceptual distortion metric which lim- 
its how ^distorted" the video is made to be : thereby 
establishing a distortion floor, the maximum amount of 
encoding, i.e., the degree of fineness to which the video 
signal is encoded, can be limited. As shown in FIG. 4, 
30 over time the distortion level tends to decrease. Howev- 
er, when the distortion floor is reached, the degree of 
fineness of the encoding associated with the distortion 
level is maintained until the video image changes or 
there is an increase in scene complexity, at which point 
3S the distortion level is allowed to increase in order to ac- 
commodate the resulting increase in the coded bit rate. 

A consequence of establishing a floor on the distor- 
tion level is that over time, the buffer fullness will de- 
crease. In accordance with the present invention, the dis- 
40 tortion floor, which corresponds to predetermined, 
non-zero level of perceptual distortion in the video signal, 
is set at such a level that the buffer will regularly achieve 
an underflow condition. During the underflow condition, 
auxiliary data from auxiliary data buffer 245 are transmit - 
ted and transmission of the video signals is suspended. 
When the buffer fills above a predetermined level, trans- 
mission of auxiliary data is suspended and transmission 
of the video signals is resumed. 

By creating artificial underflow conditions in this way, 
50 the amount of auxiliary data which may be transmitted is 
increased over the prior art. In particular, at various 
points in the encoding of the video signal, such as during 
"stills" and periods of relatively slow movement, and/or 
when particular types of video signals are being encod- 
es ed. such as 24- or 30-frame/second film, the full data rate 
capability of the channel may not be taken up by the 
number of bits being supplied to. and ultimately read out 
of. video data buffer 21 5. The system thus has the ability 
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during such periods to accommodate the transmission 
of auxiliary data that would not otherwise be able to be 
accommodated. 

Specifically, then, the system receives an auxiliary 
data bit stream on lead 240. The bits of this data stream 
are buffered in auxiliary data buffer 245 whose output 
feeds into multiplexer 225. In accordance with one em- 
bodiment, the system may provide a minimum constant 
bit-rate capability to establish a minimum bit-rate of aux- 
iliary data by dedicating a portion of the video channel 
exclusively to the transmission of auxiliary data. That is, 
bits from auxiliary data buffer 245 are continuously read 
therefrom at at least a predetermined minimum rate and 
multiplexed with the HDTV-encoded signal for presenta- 
tion to channel 220. However, as described above, when 
the video data buffer is empty--or, perhaps in other em- 
bodiments, almost empty-indicating that the channel ca- 
pacity is not being fully utilized by the current combined 
level of bits being output by HDTV encoder 21 0 and aux- 
iliary data buffer 245. multiplexer 225 increases the rate 
at which it takes auxiliary data bits from buffer 245. Later, 
as video data buffer 21 5 begins to fill, the rate of auxiliary 
data taken from data buffer 245 is decreased -ultimately 
to the aforementioned minimum constant bit rate. 

FIG. 3 further illustrates a more detailed block dia- 
gram of the HDTV receiver 30. At the far end of the chan- 
nel, the composite video/auxiliary data signal is demul- 
tiplexed by demultiplexer 305. The received HDTV video 
signal is applied to video data buffer 310, then to HDTV 
decoder 315, and ultimately to a display 320 while the 
received auxiliary data is applied to auxiliary data buffer 
325 and then to a storage device 330. That storage de- 
vice 330 might be the display 320 on which the HDTV 
signal is displayed or could be a virtual set-top box which 
can transmit the data to any kind of end-use device. 

The broadcast industry has indicated that digital 
data services will be a valuable part of HDTV. Advanta- 
geously, this arrangement provides a useful way of de- 
livering such services. In particular, the ability to transmit 
auxiliary data along with the HDTV signal provides the 
capability of delivering such services as electronic news- 
papers and magazines, electronic catalogues, paging 
services, stock quotes and program guides, as well as 
additional audio and/or video channels. 

It will be appreciated that those skilled in the art will 
be able to devise numerous and various alternative ar- 
rangements which, although not explicitly shown or de- 
scribed herein, embody the principles of the invention 
and are within its scope. 



Claims 

1 . A method comprising the steps of: 

encoding an input video signal to generate a 
variable- bit -rate encoded signal having a selected 
level of distortion, 

applying said video signal to a buffer, said level 



of distortion being selected as a function of the full- 
ness of said buffer, 

reading said video signal from said buffer and 
applying it to a transmission channel at a predeter- 
mined video output rate, and 

applying portions of an auxiliary signal to said 
channel at times when said buffer is in an underflow 
condition. 

said selected level of distortion having a floor 
which corresponds to a predetermined, non-zero 
level of perceptual distortion in said video signal. 

2. The method of claim 1 wherein said predetermined, 
non-zero level of perceptual distortion is such that 
underflow condition occurs often enough that said 
auxiliary signal can be applied to said channel at at 
least a predetermined auxiliary data rate. 

3. The method of claim 1 wherein said step of encoding 
an input video signal comprises the step of: 

generating a perceptual distortion metric indic- 
ative of a desired level of distortion for the encoded 
signal, said perceptual distortion metric being a 
function of the fullness of said buffer. 

4. A method comprising the steps of: 

encoding an input video signal to generate a 
variable-bit-rate encoded signal, said encoding 
including the steps of generating a perceptual dis- 
tortion metric indicative of a desired level of distor- 
tion for the encoded signal and encoding said input 
video signal in such a way as to achieve said desired 
level of distortion, 

applying said video signal to a buffer, said per- 
ceptual distortion metric being a function of the full- 
ness of said buffer 

reading said video signal from said buffer and 
applying it to a transmission channel at a predeter- 
mined video output rate, and 

applying portions of an auxiliary signal to said 
channel at times when said buffer is in an underflow 
condition. 

wherein said perceptual distortion metric has 
a floor whose value is such that the application of 
said auxiliary signal to said channel occurs at a pre- 
determined average bit rate. 

5. The method of claim 4 wherein said floor is a function 
of the level of distortion of said video signal at which 
a lower level of distortion will not be perceived by a 
human viewer. 

6. The method of claim 4 wherein said floor is substan- 
tially at a level of distortion is not perceived by a 
human viewer. 

7. A method of incorporating an auxiliary signal with a 
variable rate primary signal comprising the steps of: 
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encoding the primary signal at a particular 
level of quantization: applying said primary encoded 
signal to a buffer; 

transmitting said primary signal over a trans- 
mission channel at a predetermined output rate: $ 

setting the level of quantization so that an 
underflow condition is achieved in said buffer: and 

transmitting the variable auxiliary data stream 
over said transmission channel during said under- 
flow/condition. io 

8. The method according to claim 7 wherein said step 
of setting the level of quantization comprises the 
steps of: 

establishing a perceptual distortion metric is 
which is responsive to the state of buffer fullness; 

measuring a distortion level associated with 
the primary signal: and 

establishing a floor for the distortion level by 
lowering the perceptual distortion metric, thereby 20 
causing the buffer to achieve an underflow condi- 
tion. 

9. The method according to claim 3 wherein said step 

of establishing a floor comprises adjusting the per- 25 
ceptual distortion metric so that the buffer empties 
more frequently resulting in more frequent buffer 
underflow conditions and more frequent transmis- 
sion of the auxiliary signals. 
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